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Research? 


The What Works Clearinghouse (WWC) identified five studies of 
summer counseling that both fall within the scope of the Transition 

to College topic area and meet WWC group design standards. All five 
studies meet WWC group design standards without reservations, and 
no studies meet WWC group design standards with reservations. Together, these studies included 13,614 recent 
high school graduates in 10 locations. 
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According to the WWC review, the extent of evidence for summer counseling on the postsecondary outcomes of 
recent high school graduates was small for the credit accumulation and persistence domain and medium to large 
for the college access and enrollment domain. No studies meet WWC group design standards in the 12 other 
domains, so this intervention report does not report on the effectiveness of summer counseling for those domains.° 
(See the Effectiveness Summary on p. 7 for more details of effectiveness by domain.) 


Effectiveness 


Summer counseling had potentially positive effects on credit accumulation and persistence and mixed effects 
on college access and enrollment for recent high school graduates. 
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Table 1. Summary of findings* 


Improvement index (percentile points) 


Number of Number of Extent of 
Outcome domain Rating of effectiveness Average Range studies students evidence 
Credit accumulation Potentially positive effects +6 +6 1 1,397 Small 
and persistence 
College access Mixed effects +5 0 to +13 5 13,614 Medium to large 


and enrollment 
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Intervention Information 
Background 


Even after students have applied and have been accepted to college, they must complete a number of steps to 
successfully matriculate in college. For example, colleges typically require students to take placement exams, enroll 
for classes, finalize financial aid arrangements, have health insurance, complete housing and medical forms, and 

fill out other paperwork over the summer months. For first-generation college students who no longer have access 
to high school guidance counselors and who do not have family members familiar with the matriculation process, 
these requirements may seem daunting. Without a support system to help them matriculate in college, students 
may fail to complete all necessary paperwork and requirements and fail to enroll in college during the fall, which 

is a phenomenon called the “summer melt.”° 


Intervention details 


Summer counseling is designed to help college-intending high school graduates complete the steps needed to 
matriculate in college. The goal of summer counseling is to reduce the number of students who fail to matriculate 
in college, particularly among disadvantaged students. Summer counseling provides students with information 

and support for completing the steps needed to attend college, such as enrolling in courses, completing housing 
forms, registering for orientation, finalizing arrangements to secure financial aid, and addressing socioemotional 
barriers that may inhibit students from enrolling in higher education. Summer counseling involves outreach by peer 
mentors or college counselors via text messaging, email, phone, in-person meetings, instant messaging, or social 
media. These services are typically set up for students in their high schools, though some programs may be based 
in colleges or nonprofit organizations. Counseling interventions take place during the summer after high school and 
typically run for 1-2 months. 


There is no standard list of summer counseling services or delivery strategies, and implementation can vary across 
locations and service providers. There are, however, commonalities among summer counseling programs reviewed by 
the WWC. In the five studies that met WWC group design standards, students were provided summer counseling for 
about 1.5 months between high school graduation and college matriculation. In all studies, participating students were 
high school graduates who intended to enroll in college in the fall. The interventions examined in these five studies 
targeted either low-income students (two studies), underrepresented students (one study), and students from large, 
urban school districts (two studies). Summer counseling was provided by staff based in high schools (five studies), at 
a nonprofit organization (two studies), and at a college (one study), with three studies employing counselors housed 

in more than one setting. One study included a text messaging campaign, one used peer mentors (near-aged college 
students in good academic standing), and the remaining relied primarily on college counselors (note that some studies 
used more than one approach). Counseling and outreach was provided using a number of communication methods, 
including phone (five studies), in-person consultations (five studies), email (four studies), text messaging (four studies), 
social media (two studies), and instant messaging (two studies). The frequency of contacts with college-intending 
students ranged from once every 5 days to once every 2 weeks. Counseling included financial aid information 

and informational barriers in all interventions studied and socioemotional barriers to college attendance in two of the 
interventions studied. Across all five studies, an average of 58% of students met with a counselor either in person or 
virtually at least one time during the summer after high school graduation. 
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Cost 
Because summer counseling does not have a standard list of services or delivery strategies, implementation 
costs can vary across sites. Four of the five studies reviewed that met WWC group design standards without 
reservations included information about the cost of summer counseling. The cost of summer counseling involv- 
ing one-on-one assistance ranged from $100-$200 per student (as cited in Castleman, Arnold, & Wartman, 2012; 
Castleman, Page, & Schooley, 2014; and Castleman, Owen, & Page, 2015a). The cost of summer counseling via 
text messaging was $7 per student for the text messaging campaign (and any associated counseling that came 
out of the text messaging outreach), and about $2 per student for just the text messaging portion of the interven- 
tion (as cited in Castleman & Page, 2015). Summer counseling via a peer mentoring intervention cost about $80 
per student to pay the salaries of 20 peer mentors who worked 20 hours per week and for supervising advisors 
(as cited in Castleman & Page, 2015). 
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Research Summary 


The WWC identified six eligible studies that investigated the effects of Table 2. Scope of reviewed research 
summer counseling on college enrollment and persistence of recent 


Grades 12-PS 
high school graduates. An additional 4 studies were identified but do ar iad RaNiaual 
not meet WWC eligibility criteria (see the Glossary of Terms in this doc- eet A acl ae 

Program type Supplement 


ument for a definition of this term and other commonly used research 
terms) for review in this topic area. Citations for all 10 studies are in the 
References section, which begins on p. 9. 


The WWC reviewed six eligible studies against group design standards. Five studies are randomized controlled 
trials that meet WWC group design standards without reservations. This report summarizes those five studies. 
The remaining study does not meet WWC group design standards. 


Summary of studies meeting WWC group design standards without reservations 


Castleman et al. (2012) conducted a randomized controlled trial to examine the effectiveness of college counseling 
delivered to students during the summer after high school graduation. In seven small high schools in Rhode Island, 
all 162 graduating seniors from the class of 2008 were assigned to receive college counseling during the summer or 
to a comparison group that did not receive the intervention. The summer college counseling services aimed to help 
students complete financial aid applications, address any gaps between financial aid packages and anticipated 
costs of attending college, or any other socioemotional issues that need to be addressed before matriculating 

into college. Counseling was provided via in-person consultations, email, and text messages. The WWC based its 
effectiveness rating on college enrollment outcomes among 80 students in the intervention group and 82 students 
in the comparison group. 


Castleman et al. (2014) conducted a randomized controlled trial in Boston (MA) and Fulton County (GA) schools to 
examine the effectiveness of college counseling over the summer months after high school graduation. Students 
from six high schools in the Fulton County site were eligible for the study if they planned to pursue postsecondary 
education, if they had applied to college, and if they had been accepted by at least one institution. Students in the 
Boston site were all applicants to the Last Dollar Scholarship, which provides Boston students with funds to pay 
unmet college costs. Eligible students (all from the class of 2011) were then randomly assigned to receive summer 
counseling or to serve in a comparison group. Most consultations in Boston took place in person, while most con- 
sultations in Fulton County took place over the phone. The WWC based its effectiveness rating on college enroll- 
ment and persistence outcomes for the pooled student sample, which included 886 students in the intervention 
group and 1,487 students in the comparison group. 


Castleman et al. (2015a) conducted a randomized controlled trial to examine the effectiveness of counselor out- 
reach support on high school graduates’ enrollment in college. High school graduates from the class of 2012 

in New Mexico who had been accepted into college were eligible to participate in the study and were randomly 
assigned to the intervention group or to a comparison group. The goal of the intervention was to increase the 
number of students entering college by providing assistance with summer tasks relating to enrolling in college 
(e.g., finalizing financial aid, completing the FAFSA, arranging on-campus housing, signing up for placement tests, 
selecting classes, organizing transportation to campus). Counseling was provided in person and over the phone by 
eight counselors based at the University of New Mexico and 13 counselors based in Albuquerque Public Schools. 
The WWC based its effectiveness rating on college enrollment outcomes among 1,074 students in the intervention 
group and 528 students in the comparison group. 
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Castleman et al. (2015b) conducted a cluster randomized controlled trial to examine the effectiveness of an online 
software (Bridgit) that can be used by high school counselors to track college-intending students’ progress on sum- 
mer tasks relating to college enrollment. The goal of the Bridgit platform was to track progress on pre-matriculation 
tasks, help counselors assist with these tasks, and facilitate communication between students and counselors, par- 
ticularly for those students at greater risk of failing to matriculate. Communication between counselors and students 
took place via text messages that could be sent through the Bridgit platform, or by phone if additional follow up was 
needed. Schools in Tennessee and Missouri serving at least 60% of students eligible for free/reduced-price lunch 
within each district were paired based on the prior year’s college enrollment rate. In summer 2014, one school was 
randomly assigned in each pair into the intervention (Bridgit) group, with the remaining schools assigned to the 
comparison group, resulting in six intervention schools and six comparison schools. Counselors provided assis- 
tance via text messaging. The WWC based its effectiveness rating on college enrollment outcomes among 1,761 
students in the intervention schools and 1,520 students in the comparison schools. 


Castleman and Page (2015) conducted a randomized controlled trial to examine the effectiveness of two summer 
counseling interventions to help support high school graduates in their transition to postsecondary education. High 
school graduates in Texas, Massachusetts, and Pennsylvania from the class of 2013 who intended to go to college 
were eligible to participate in the study. Students were randomly assigned to a text message group, a peer mentor- 
ing group, or a comparison group. The WWC based its effectiveness rating on college enrollment outcomes for the 
two types of interventions. The texting intervention portion of the study included 2,524 students who were randomly 
assigned to a text message group and 2,535 students in the comparison group. The peer mentoring portion of the 
study included 934 students who were randomly assigned to a peer mentoring group and 1,272 students who were 
randomly assigned to a comparison group. 


Summary of studies meeting WWC group design standards with reservations 
No studies of summer counseling met WWC group design standards with reservations. 
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Effectiveness Summary 


The WWC review of summer counseling for the Transition to College topic area includes student outcomes in 14 domains: 
general academic achievement (middle school), general academic achievement (high school), attendance (middle 
school), attendance (high school), college readiness, staying in high school, progressing in high school, completing 
high school, college access and enrollment, college attendance, credit accumulation and persistence, general 
academic achievement (college), degree attainment (college), and labor market. The five studies of summer coun- 
seling that met WWC group design standards reported findings in two of the 14 domains: (1) credit accumulation 
and persistence and (2) college access and enrollment. The following findings present the authors’ estimates and 
WWC-calculated estimates of the size and statistical significance of the effects of summer counseling on recent 
high school graduates. Additional comparisons are available as supplemental findings in Appendix D. The supple- 
mental findings do not factor into the intervention’s rating of effectiveness. For a more detailed description of the 
rating of effectiveness and extent of evidence criteria, see the WWC Rating Criteria on p. 28. 


Summary of effectiveness for the credit accumulation and persistence domain 


Table 3. Rating of effectiveness and extent of evidence for the credit accumulation and persistence domain 


Rating of effectiveness Criteria met 


Potentially positive effects In the one study that reported findings, the estimated impact of summer counseling on college persistence 
Evidence of a positive effect with outcomes was statistically significant, but not substantively important. 
no overriding contrary evidence. 


Extent of evidence Criteria met 


Small One study that included 1,397 students located in six high schools in Fulton County and an unspecified number 
of high schools in Boston reported evidence in the credit accumulation and persistence domain. 


One study that met WWC group design standards without reservations reported findings in the credit accumulation 
and persistence domain. 


Castleman et al. (2014) reported, and the WWC confirmed, a positive and statistically significant difference between 
the summer counseling group and the comparison group on the percentage of students who remained enrolled in 
college after the spring of their freshman year and after the fall of their sophomore year. The WWC characterizes 
this finding as a statistically significant positive effect. 


Thus, for the credit accumulation and persistence domain, one study of summer counseling found a statistically 
significant positive effect. This results in a rating of potentially positive effects, with a small extent of evidence. 


Summary of effectiveness for the college access and enrollment domain 


Table 4. Rating of effectiveness and extent of evidence for the college access and enrollment domain 


Rating of effectiveness Criteria met 
Mixed effects In the five studies that reported findings, the estimated impact of the intervention on outcomes in the college 
Evidence of inconsistent effects. access and enrollment domain was positive and statistically significant in one study, positive and substantively 


important in one study, and indeterminate in three other studies. 
Extent of evidence Criteria met 


Medium to large Five studies that included 13,614 students in 10 locations reported evidence of effectiveness in the college 
access and enrollment domain. 
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Five studies that met WWC group design standards without reservations reported findings in the college access 
and enrollment domain. 


Castleman et al. (2012) reported, and the WWC confirmed, no statistically significant difference between interven- 
tion and comparison group students’ fall college enrollment rates (whether full-time or part-time) and full-time 
enrollment rates. The effect size for both of these comparisons was large enough to be considered substantively 
important. The WWC characterizes this study finding as a substantively important positive effect. 


Castleman et al. (2014) reported, and the WWC confirmed, a positive and statistically significant difference between 
the intervention group and the comparison group on the percentage of students who enrolled in college in the fall 
after high school graduation. The WWC characterizes this finding as a statistically significant positive effect. 


Castleman et al. (2015a) reported, and the WWC confirmed, no statistically significant difference between the inter- 
vention group and the comparison group on fall college enrollment. The WWC characterizes this study finding as an 
indeterminate effect. 


Castleman et al. (2015b) reported, and the WWC confirmed, no statistically significant difference between the 
summer counseling group and the comparison group on fall college enrollment. The WWC characterizes this study 
finding as an indeterminate effect. 


Castleman and Page (2015) reported, and the WWC confirmed, no statistically significant difference between the 
intervention groups and the comparison groups on fall college enrollment. The WWC characterizes this study find- 
ing as an indeterminate effect. 


Thus, for the college access and enrollment domain, one study of summer counseling found a statistically significant 
positive effect, one study of summer counseling found a substantively important positive effect, and three studies 
found indeterminate effects. This results in a rating of mixed effects, with a medium to large extent of evidence. 
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Appendix A.1: Research details for Castleman et al. (2012) 


Castleman, B. L., Arnold, K., & Wartman, K. L. (2012). Stemming the tide of summer melt: An experi- 
mental study of the effects of post-high school summer intervention on low-income students’ 
college enrollment. Journal of Research on Educational Effectiveness, 5(1), 1-17. Retrieved from 
https://eric.ed.gov/?id=EJ952097 


Additional source: 

Arnold, K., Fleming, S., DeAnda, M., Castleman, B. L., & Wartman, K. L. (2009). The summer 
flood: The invisible gap among low-income students. Thought and Action, Fall, 23-34. 
Retrieved from https://eric.ed.gov/?id=EJ930460 


Table A1. Summary of findings Meets WWC group design standards without reservations 
Study findings 


Average improvement index 
Outcome domain Sample size (percentile points) Statistically significant 


College access and enrollment 162 high school graduates +13 No 


Setting |The study was conducted with the entire 2008 graduating class from seven urban schools in 
Rhode Island. These schools were in a network of charter schools run by Big Picture Learning. 
Schools in the Big Picture network are small learning communities that provide individualized 
curricula, internship programs, and focus on relationship building by maintaining a 4-year peer 
group and assigning teacher advisors to students. Every Big Picture student is expected to 
take college entrance examinations, apply to college, and complete financial aid applications. 


Study sample —_ Within each of the seven Big Picture schools, students who were in the graduating class of 
2008 were randomly assigned to the intervention group (n = 80) or to the comparison group 
(n = 82). In the intervention group, 43% of students were male, 49% were Hispanic, 29% were 
Black, 21% were White, and 2% were Asian. Moreover, 17% of intervention group students 
had an IEP, 68% received free lunch, and 81% reported that they planned to attend college as 
of graduation. 


In the comparison group, 45% of students were male, 40% were Hispanic, 29% were Black, 
21% were White, and 5% were Asian; 17% of students had an IEP, 62% received free lunch, 
and 80% reported that they planned to attend college as of graduation. 


Intervention Recent high school graduates worked with two school-based counselors throughout the sum- 
group mer of 2008. The primary goal of the counseling program was to help students secure financial 

aid and address any gaps between financial aid packages and the anticipated costs of attend- 
ing college. Counselors also acted as a liaison to the colleges and addressed any information 
barriers the students faced, and helped students address socioemotional barriers to enroll- 
ment (e.g., students having second thoughts about attending college, dealing with pressure 
from family or friends to stay home). The counselors took an active role in communicating with 
students via emails, text messages, and in-person consultations. The majority of the inter- 
vention group (84%) met with a counselor at some point during the summer. Counseling was 
provided from June 15, 2008 through September 1, 2008. 
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Comparison Students in the comparison group had access to regular counseling services delivered by 
group the high school; however, comparison students did not receive proactive counseling from the 
counselors over the summer. The authors reported that 21% of the students in the comparison 
condition contacted a counselor during the summer intervention period. 


Outcomes and Authors reported five outcomes. Two outcomes presented as primary findings in Appendix C 
measurement include college enrollment in the fall following high school graduation, and full-time college enroll- 
ment. Both of these variables are binary and were collected from the National Student Clearinghouse. 


Two outcomes are reported as supplementary findings in Appendix D: enrolled at a 4-year institu- 
tion and enrolled at a 2-year institution. These outcomes were also collected from the National 
Student Clearinghouse and are dichotomous measures of three possible outcomes (1) enrollment 
in a 2-year institution, (2) enrollment in a 4-year institution, and (3) not enrolled in college. For 
example, 2-year enrollment is answered “yes” for a student who enrolled in a 2-year institution 
and “no” for students who enrolled in a 4-year institution or who did not enroll. Because these 
findings provide both positive and negative outcomes in the same category, they were presented 
as supplemental findings. These supplemental findings do not factor into the intervention’s rating 
of effectiveness. For a more detailed description of these outcome measures, see Appendix B. 


Study authors also reported on whether students enrolled in the institution they intended to 
enroll in at high school graduation. This outcome was used to examine whether students fol- 
lowed through on their expected plans after high school and is not eligible for review under the 
Transition to College protocol. 


Support for Two college counselors were hired full-time to work through the summer to provide counseling 
implementation _ to students. 


Appendix A.2: Research details for Castleman et al. (2014) 


Castleman, B. L., Page, L. C., & Schooley, K. (2014). The forgotten summer: Does the offer of college 
counseling after high school mitigate summer melt among college-intending, low-income high 
school graduates? Journal of Policy Analysis and Management, 33(2), 320-344. Retrieved from 
https://eric.ed.gov/?id=EJ1027721 


Table A2. Summary of findings Meets WWC group design standards without reservations 
Study findings 
Average improvement index 
Outcome domain Sample size (percentile points) Statistically significant 
College access and enrollment 2,373 high school graduates +5 Yes 
Credit accumulation 1,397 high school graduates +6 Yes 


and persistence 


Setting | The study was conducted in Boston, MA and in Fulton County Schools (FCS) in the metro 
Atlanta, GA area. In Boston, the counseling sessions took place primarily at the provider’s 
(uAspire) Center for College Affordability in Boston. In Fulton County, most counseling took 
place over the phone rather than face-to-face. 
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Study sample There were 2,373 students in the overall sample, including 886 students in the intervention group 
and 1,487 students in the comparison group. 


In Boston, high school students who were participants in uAspire’s High School Advising Program 
and who applied for uAspire’s Last Dollar Scholarships were included in the study sample. Study 
authors assigned each of the 927 applicants to a team of 11 advisors, matching applicants to teams 
with the advisor who had worked with them before whenever possible. Study authors then ran- 
domly assigned students to the intervention group (n = 406) or the comparison group (n = 521). The 
intervention took place in Boston from June 27, 2011 to August 10, 2011. 


In Fulton County, study authors selected six high schools with the highest estimated rates of “sum- 
mer melt.” Within these schools recent high school graduates were identified who indicated on the 
Fulton County Schools Senior Exit Survey that they (a) were planning to attend college the following 
fall, (b) had applied to at least one postsecondary institution, and (c) had been accepted to at least 
one postsecondary institution. In order to meet the researchers’ target of 80 students served per 
school, eligibility criteria were expanded in two schools to include students waiting to hear about 
their acceptance. With 1,446 students identified as eligible for the study, researchers then randomly 
selected 80 students in each school to receive the intervention (n = 480) and the remaining students 
were assigned to the comparison group (n = 966). In Fulton County, the intervention took place 
between June 6, 2011 and July 11, 2011. 


Across both sites, ethnic minority students comprised 72% of the sample (43% Black, 13% 
Hispanic, 10% Asian, 5% multiracial, and 2% other race/ethnicity), and 58% of the students were 
female. Eighty-one percent of students completed the FAFSA. Of those who completed the FAFSA 
in Boston, 62% had an Expected Family Contribution (EFC) of zero and another 23% had an EFC 
that was nonzero, but still within the range of Pell grant eligibility. In Fulton County, 37% of students 
qualified for free or reduced-price lunch (FRL). 


Intervention Counselors attempted to contact each student in the intervention group via phone, email, text, and 
group Facebook to offer support. In Fulton County, counselors were encouraged to use an intake form 

that outlined the tasks required for college enrollment in their initial meeting with students. Counsel- 
ors who met with students in person primarily did so at the school from which they were working, 
but the counselors depended on phone conversations to provide most of their support. In Boston, 
upon reaching students, advisors offered students a $25 gift card to attend an in-person meet- 
ing. During the first in-person meeting, counselors completed a college assessment protocol that 
included the following elements: (1) review of the student’s financial aid award letter and guidance 
on financial aid tailored to the amount of unmet need; (2) discussion of the calendar of summer 
deadlines at the college the student planned to attend, and help with understanding and completing 
paperwork from the college; and (3) assessment of any social or emotional barriers to college enroll- 
ment faced by the student. After the assessment, counselors and students developed a list of tasks 
that needed completion before starting college in the fall. Counselors followed up with students 
individually to check on their progress. After the initial meeting, counselors and students commu- 
nicated mostly via phone, email, and text, though counselors also conducted in-person follow-up 
meetings with some students. 
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Comparison 
group 


Outcomes and 
measurement 


Support for 
implementation 


Across both sites, 52% of students had any communication with a counselor, and in Boston, 52% 
of students had at least one face-to-face meeting with an advisor. In Atlanta, approximately 25% 

of the non-FRL students had contact with a counselor, while nearly 54% of the FRL students had 
contact with a counselor. Authors noted that many of the counselors’ interactions with students 
focused on issues of financial aid. Counselors also reported addressing a variety of informational 
questions, such as how to access a college’s web portal, how to complete required paperwork, and 
what the college matriculation process entailed. 


The comparison group students did not receive outreach though they were assigned to a counselor. 
Counselors were instructed not to deny support to any comparison group student who actively 
sought help. According to logs maintained by the counselors, about 1% of the comparison group 
students had contact with an advisor. 


College enrollment is the primary outcome in this study. Enrollment data were obtained from the 
National Student Clearinghouse for the fall of 2011 (at the end of the fall semester of the student’s 
freshman year), spring of 2012 (at the end of the spring semester of the student’s freshman year), 
and fall of 2012 (at the end of the first semester of the student’s sophomore year). The initial fall 
2011 enrollment measure falls under the college access and enrollment domain, while the continued 
enrollment or persistence outcomes (spring 2012 and fall 2012) fall under the credit accumulation 
and persistence domain. 


Subgroup analyses were presented for socioeconomic status subgroups, as defined by free or 
reduced-price lunch status for students in the Atlanta site and by expected family contribution 
and Pell Grant eligibility for students in the Boston site. The supplemental findings are reported in 
Appendix D and do not factor into the intervention’s rating of effectiveness. For a more detailed 
description of these outcome measures, see Appendix B. 


The study also examined whether students enrolled and persisted at (1) the specific institution in 
which they intended to enroll as of high school graduation and (2) the type of institution (i.e., 2-year 
vs. 4-year, public vs. private) in which they intended to enroll as of high school graduation. These 
outcomes focused on intentions after high school and are not eligible for review under the Transition 
to College protocol. 


In Atlanta, the study authors provided supplemental training for the counselors that focused 
on the federal and state financial aid application process. In Boston, the study authors pro- 
vided the uAspire counselors with a protocol for their outreach activities and supplied the 
assessment protocol that guided the counselors’ advising. 
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Appendix A.3: Research details for Castleman et al. (2015a) 


Castleman, B. L., Owen, L. & Page, L. C. (2015a). Stay late or start early? Experimental evidence on the 
benefits of college matriculation support from high schools versus colleges. Economics of Educa- 
tion Review, 47, 168-179. doi: 10.1016/j.econedurev.2015.05.010 


Table A3. Summary of findings Meets WWC group design standards without reservations 
Study findings 


Average improvement index 
Outcome domain Sample size (percentile points) Statistically significant 


College access and enrollment 1,602 high school graduates 0 No 


Setting The study was conducted in Albuquerque, New Mexico. Summer outreach counseling was 
based in two settings: Albuquerque Public Schools (APS) high schools and the University of 
New Mexico (UNM). 


Study sample The study included a total of 1,602 students, including 539 students who received summer 
counseling from counselors based at UNM, 535 students who received summer counseling 
from counselors based at APS high schools, and 528 students who did not receive summer 
counseling. Participants were APS high school graduates from the class of 2012 who had 
been admitted to UNM. Across the full sample, 41% of students were male, 50% were His- 
panic, 2% were Black, 5% were Asian, 85% were White, 1% were English Learners, and 29% 
were eligible for free or reduced-price lunch. The average high school GPA for the full sample 
was 3.26. 


Intervention Study authors hired 21 school counselors, placing eight counselors at UNM and 13 counselors 
group at high schools in APS. Counselors reached out to students via phone, email, and text mes- 

sage to help students complete summer tasks relating to enrolling in college (e.g., finalizing 
financial aid, completing the FAFSA, arranging on-campus housing, signing up for placement 
tests, selecting classes, and organizing transportation to campus). Counseling support was 
provided in person and over the phone. Counselors had caseloads between 60 and 100 stu- 
dents, and worked 10-20 hours per week for 5-6 weeks during the summer of 2012. Just over 
52% of students in the intervention group met with a counselor over the summer, including 
50% of Hispanic students and 51% of female students. 


Comparison Students in the comparison group did not receive outreach support from an intervention- 
group trained counselor. Less than 1% of students in the comparison group met with a counselor in 
the same time period. 
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Outcomes and Study authors reported findings on college enrollment in the fall following high school gradu- 

measurement ation (i.e., fall 2012). The outcome is binary and was obtained from the National Student 
Clearinghouse. Additionally, supplemental findings were reported by ethnicity and gender (i-e., 
for Hispanic males, non-Hispanic males, Hispanic females, and non-Hispanic females). The 
authors also reported on the effects of summer counseling based in APS and in UNM. These 
supplemental findings are reported in Appendix D and do not factor into the intervention’s 
effectiveness. For a more detailed description of these outcome measures, see Appendix B. 


The study also examined the impact of summer counseling on enrollment at UNM, as well as 
enrollment at a college other than UNM in the fall after graduation. These outcomes are not 
eligible for review under the Transition to College topic area, since they focus on enrollment at 
a specific institution. 


Support for Prior to the start of the intervention, counselors were trained on how to review required college 
implementation paperwork and were given tools to guide their interactions with students (e.g., checklists). 
Additionally, college-based counselors also were provided a day-long on-campus orientation 
covering university-specific details on required summer tasks for students, and a staff liaison 
was available at UNM to answer counselors’ questions. 


Appendix A.4: Research details for Castleman et al. (2015b) 


Castleman, B. L., Owen, L. & Page, L. C. (2015b). Report to College Bound St. Louis on the implementation 
and impact of the 2014 summer melt intervention utilizing Bridgit. St. Louis, MO: College Bound. 


Table A4. Summary of findings Meets WWC group design standards without reservations 
Study findings 


Average improvement index 
Outcome domain Sample size (percentile points) Statistically significant 


College access and enrollment 3,281 high school graduates +3 No 


Setting | The study was conducted in Shelby County Schools in Tennessee and in the Ferguson-Floris- 
sant School District in Missouri. 


Study sample _A total of 12 schools were included in the study, with 10 schools from Shelby County and 
two schools from Ferguson-Florissant. Schools had to meet three criteria to participate in the 
study: (1) at least 60% of students had to be eligible for free/reduced-price lunch; (2) at least 
50% of the graduating seniors had to have submitted a FAFSA form by June 2014; and 
(3) at least 100 students had to have completed the FAFSA by June 2014. Eligible schools 
within each district were then paired based on the prior year’s college enrollment rate. In sum- 
mer 2014, one school was randomly assigned in each pair into the intervention (Bridgit) group, 
with the remaining school assigned to the comparison group, resulting in six intervention 
schools and six comparison schools. 
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Intervention 
group 


Comparison 
group 


Outcomes and 
measurement 


Support for 
implementation 


Within these 12 schools, 3,281 students participated in the study, including 1,761 students in 
the intervention group and 1,520 students in the comparison group. Students who indicated 
on a spring 2014 survey that they were college-intending at the time of high school graduation 
were included in the study. Across both conditions, 59% of students were eligible for free or 
reduced-price meals; 86% were Black, 6% were Hispanic, and 5% were White; 10% had an 
IEP; and 53% were female. These students had an average ACT score of 16.67 and an aver- 
age GPA of 2.50. 


The Bridgit platform was implemented in intervention schools during the summer of 2014. 

The Bridgit platform is a schoolwide intervention used by school counselors to track students’ 
progress toward college enrollment. Students first created a profile in the system that identi- 
fied which college they planned to attend and where they were in the admissions process. 

The system then prioritized pre-matriculation steps needed that were tailored to the student’s 
chosen college. Counselors or college advisors were assigned to each student, and monitored 
the prioritized list of pre-matriculation tasks for each of their students. Counselors were able to 
communicate with students via text messages directly in the Bridgit platform, and if additional 
follow up was needed they could place personal phone calls to students. The Bridgit platform 
collected data about counselor outreach to students, whether counselors were able to make 
contact, and whether that contact resulted in students updating their profiles and/or in-person 
meetings. Over 60% of students in the intervention group interacted with a counselor, primarily 
via text messaging and phone. 


Students in comparison schools participated in the “business-as-usual” procedures for gradu- 
ating seniors with regard to support for transitioning to college. 


The authors measured one primary outcome: college enrollment in the fall following high 
school graduation. The outcome is binary and was obtained from the National Student Clear- 
inghouse. Additionally, supplemental findings were reported for enrollment in 2-year institu- 
tions and enrollment in 4-year institutions. The supplemental findings are reported in Appendix 
D and do not factor into the intervention’s effectiveness. For a more detailed description of 
these outcome measures, see Appendix B. 


A 2-day training was conducted for counselors using the Bridgit system. School counselors 
implementing Bridgit in intervention schools provided feedback to the study authors on the 
challenges and advantages of the tool during the course of the study. 
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Appendix A.5: Research details for Castleman and Page (2015) 


Castleman, B. L., & Page, L. C. (2015). Summer nudging: Can personalized text messages and peer men- 
tor outreach increase college going among low-income high school graduates? Journal of Eco- 
nomic Behavior and Organization, 115, 144-160. Retrieved from https://eric.ed.gov/?7id=EJ1124459 


Additional sources: 

Castleman, B. L. (2013). Assistance in the 11th hour: Experimental interventions to mitigate 
summer attrition among college-intending high school graduates (Doctoral dissertation). 
Available from ProQuest Dissertations and Theses database. (UMI No. 3662580) 


Castleman, B. L., & Page, L. C. (2013). The not-so-lazy days of summer: Experimen- 
tal interventions to increase college entry among low-income high school gradu- 
ates. New Directions for Youth Development, 140, 77-97. Retrieved from https://eric. 
ed.gov/?id=EJ1031658 


Table A5. Summary of findings Meets WWC group design standards without reservations 
Study findings 


Average improvement index 
Outcome domain Sample size (percentile points) Statistically significant 


College access and enrollment 6,196 high school graduates +2 No 


Setting This study examined two summer counseling interventions across five different research sites in 
Massachusetts (Boston, Lawrence, and Springfield), Pennsylvania (Philadelphia), and Texas (Dallas). 


Study sample = Across all five study sites, 2012 high school graduates who were “college-intending” were eligible 
for study participation. The definition of college-intending students varied by site. In Philadelphia, 
college-intending students were identified via a high school exit survey; in uAspire sites (Boston, 
Lawrence, and Springfield), college-intending students were defined as having at least two meet- 
ings with a uAspire advisor during the student’s senior year; and in Dallas, college-intending stu- 
dents were defined as having started (or completed) the FAFSA as of high school graduation. 


In Philadelphia, students were randomly assigned to peer mentors within each of five schools, 
to ensure a caseload of 40 students per mentor (one school had two mentors). The study 
sample in Philadelphia included 240 students in the intervention group and 203 students in the 
comparison group. 


In Dallas, nine counselors were each assigned to a set of high schools within the district, and 
eligible students were then randomly assigned within each counselor’s cluster to either receive the 
texting intervention (n = 1,454) or to serve in a comparison group (n = 1,466). 


In uAspire sites in Boston, Lawrence, and Springfield, students were randomly assigned by site to 
receive peer mentoring (n = 450 in Boston, 94 in Lawrence, and 150 in Springfield), text messaging 
(n = 697 in Boston, 100 in Lawrence, and 273 in Springfield), or serve in a comparison group 

(n = 696 in Boston, 100 in Lawrence, and 273 in Springfield). 


Altogether, the peer mentoring intervention had 934 students in the intervention group and 1,272 
students in the comparison group, and the texting intervention had 2,524 students in the interven- 
tion group and 2,535 students in the comparison group. The pooled sample of 6,196 students was 
58% female; 46% Hispanic, 37% Black, 8% White, and 13% other race/ethnicity. Seventy-eight 
percent qualified for free or reduced-priced lunch, and 94% had completed the FAFSA. 


Summer Counseling March 2018 Page 18 


WWC Intervention Report 


Intervention |§ The automated text messaging campaign was implemented in Dallas, TX; Boston, MA; Law- 
group rence, MA; and Springfield, MA. During the summer of 2012, students and their parents in the 

text messaging intervention were sent a series of 10 automated text messages to remind them 
about tasks required for college enrollment and to prompt them to request additional help when 
needed. The texts included reminders to access important paperwork online, register for orienta- 
tion, register for placement tests, complete housing forms, sign up for/waive health insurance, 
and offers to help students complete the FAFSA and interpret financial aid award letters and 
tuition bills. A text message was sent approximately every 5 days between early July and mid- 
August. In Dallas, the authors collaborated with the Dallas Independent School District to link 
students to one of nine college counselors to provide additional assistance. In Boston, Law- 
rence, and Springfield, the authors collaborated with a nonprofit organization, uAspire, to link 
students to financial aid advisors at participating high schools. 


The peer mentoring intervention was implemented in Boston, MA; Lawrence, MA; Springfield, 
MA; and Philadelphia, PA. Students in the peer mentoring intervention group received contacts 
from peer mentors who assessed their readiness to matriculate into college in the fall 2012 
semester. Peer mentors discussed various topics with their mentees, including whether students 
were still planning to enroll in college, whether students had completed the FAFSA, whether 
students had received and reviewed financial aid letters, and whether students had registered 
for orientation and placement tests. Subsequent meetings and phone conversations served to 
address any other issues the students may have encountered. Counseling took place between 
mid-June and mid-August. In Boston, Lawrence, and Springfield, uAspire selected and trained 
the peer mentors who delivered the intervention. In Philadelphia, the authors collaborated with 
Mastery Charter Schools which supplied counselors from five high school campuses. 


Comparison The students in the comparison condition did not receive the intervention, but had access to 
group _ typical school supports. Two percent of the sample interacted with a counselor. 


Outcomes and One primary outcome is eligible for review under the college access and enrollment domain: 

measurement — college enrollment in the fall following high school graduation. The outcome is binary and was 
obtained from the National Student Clearinghouse. The authors also reported on two other out- 
comes: enrolled at a 4-year institution and enrolled at a 2-year institution. These supplemental 
findings are reported in Appendix D and do not factor into the intervention’s rating of effective- 
ness. For a more detailed description of these outcome measures, see Appendix B. 


Support for Signal Vine was contracted to deliver the text messages used in the text messaging campaign. 
implementation In Boston, Lawrence, and Springfield, uAspire was responsible for all peer mentor selections, 
training, and ongoing support and supervision of the mentors through the summer months. In 
Philadelphia, Mastery Charter Schools was responsible for all peer mentor selections, train- 
ings, and ongoing support and supervision of the mentors through the summer months. 
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Appendix B: Outcome measures for each domain 


Credit accumulation and persistence 


Continuous first-year enrollment Continuous first-year enrollment for the fall and spring semesters was collected from the National Student 
Clearinghouse (as cited in Castleman et al., 2014). This measure was reported in both the Boston and Atlanta 
sites as a binary outcome. 


Continuous enrollment into Continuous enrollment into sophomore year (for the first three semesters after high school) was obtained from 
sophomore year the National Student Clearinghouse (as cited in Castleman et al., 2014). This measure was reported in both the 
Boston and Atlanta sites as a binary outcome. 


College access and enrollment 


Enrollment Enrollment was measured by enrollment in college in the fall semester following high school graduation (as cited 
in Castleman et al., 2012). Enrollment data were collected from the National Student Clearinghouse and were 
reported as a binary outcome. 


Full-time enrollment Full-time college enrollment was measured by enrollment as a full-time student, with part-time enrollment and 
no enrollment included in the denominator of this measure (as cited in Castleman et al., 2012). Full-time enroll- 
ment data were collected from the National Student Clearinghouse and were reported as a binary outcome. 


Enrolled in 2-year institutions Enrollment in a 2-year institution was measured as enrollment in a 2-year institution in the fall semester 
following high school graduation (as cited in Castleman et al., 2012). Enrollment data were collected from the 
National Student Clearinghouse and were reported as a binary outcome. This outcome is only reported as a 
supplemental finding. 


Enrolled in 4-year institutions Enrollment in a 4-year institution was measured as enrollment in a 4-year institution in the fall semester 
following high school graduation (as cited in Castleman et al., 2012). Enrollment data were collected from the 
National Student Clearinghouse and were reported as a binary outcome. This outcome is only reported as a 
supplemental finding. 
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Appendix C.1: Findings included in the rating for the credit accumulation and persistence domain 


Mean 
(standard deviation) WWC calculations 
Sample Intervention Comparison Mean Effect Improvement 
Outcome measure size group group difference size index p-value 
Castleman et al., 2014 
Continuous first-year Full 1,397 82.4 78.5 3.9 0.15 +6 < .05 
enrollment (%) sample students (na) (na) 
Continuous enrollment into Full 1,397 alee 66.3 5.0 0.14 +6 < (05 
sophomore year (%) sample students (na) (na) 
Domain average for credit accumulation and persistence (Castleman et al., 2014) 0.15 +6 Statistically 


significant 


Domain average for credit accumulation and persistence across all studies 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on outcomes, representing the average change expected for all individuals who are 
given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the change in an 
average individual’s percentile rank that can be expected if the individual is given the intervention. WWC-computed average effect size is a simple average rounded to two decimal 
places; the average improvement index is calculated from the average effect size. The statistical significance of each study’s domain average was determined by the WWC. Some 
statistics may not sum as expected due to rounding. na = not applicable. 


@ For Castleman et al. (2014), a correction for multiple comparisons was needed but did not affect whether any of the contrasts were found to be statistically significant. The WWC 

did not need to make corrections for clustering or to adjust for baseline differences. The p-values presented here were reported in the original study. Means for the intervention and 
comparison groups are covariate adjusted, and reflect the pooled Boston and Fulton samples. Findings for each site are presented in Appendix D. This study is characterized as having 
a statistically significant positive effect because the estimated effect is positive and statistically significant. For more information, please refer to the WWC Procedures and Standards 
Handbook (version 3.0), p. 26. 


Appendix C.2: Findings included in the rating for the college access and enrollment domain 


Mean 
(standard deviation) WWC calculations 
Study Sample Intervention Comparison Mean Effect Improvement 
Outcome measure sample size group group difference _ size index 
Castleman et al., 2012 
Enrollment (%) Full sample 162 students 58.0 45.0 13.0 0.32 +12 <0) 
(na) (na) 
Full-time enrollment (%) — Full sample 162 students 46.0 32.0 14.0 0.36 +14 anil) 
(na) (na) 
Domain average for college access and enrollment (Castleman et al., 2012) 0.34 +13 Not statistically 
significant 
Castleman et al., 2014° 
Enrollment (%) Fullsample 2,373 students 86.0 82.7 ore 0.15 +6 < .05 
(na) (na) 
Domain average for college access and enrollment (Castleman et al., 2014) 0.15 +6 Statistically 
significant 
Castleman et al., 2015a° 
Enrollment (%) Fullsample 1,602 students 91.7 91.8 =0) —0.01 0 0 
(na) (na) 
Domain average for college access and enrollment (Castleman et al., 2015a) -0.01 0 Not statistically 
significant 
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Mean 
(standard deviation) WWC calculations 
Study Sample Intervention Comparison Mean Effect Improvement 
Outcome measure sample size group group difference _ size riti(ey¢ 
Castleman et al., 2015b® 
Enrollment (%) Fullsample 3,281 students 56.0 G20 Bu 0.09 +3 < 10 
(na) (na) 
Domain average for college access and enrollment (Castleman et al., 2015b) 0.09 +3 Not statistically 
significant 
Castleman and Page, 2015° 
Enrollment (%) Texting 5,059 students TAS 69.6 1g 0.06 +2 > 10 
intervention: (na) (na) 
Full sample 
Enrollment (%) Peer mentoring 2,206 students 69.9 67.6 2.3) 0.06 +3 0) 
intervention: (na) (na) 
Full sample 
Domain average for college access and enrollment (Castleman & Page, 2015) 0.06 +2 Not statistically 
significant 
Domain average for college access and enrollment across all studies a +5 na 


Table Notes: For mean difference, effect size, and improvement index values reported in the table, a positive number favors the intervention group and a negative number favors 
the comparison group. The effect size is a standardized measure of the effect of an intervention on outcomes, representing the average change expected for all individuals who are 
given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate presentation of the effect size, reflecting the change in an 
average individual’s percentile rank that can be expected if the individual is given the intervention. WWC-computed average effect size is a simple average rounded to two decimal 
places; the average improvement index is calculated from the average effect size. The statistical significance of each study’s domain average was determined by the WWC. Some 
statistics may not sum as expected due to rounding. na = not applicable. 


@ For Castleman et al. (2012), no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value presented here was 
reported in the original study. Means for the intervention group are covariate adjusted. This study is characterized as having a substantively important effect because the estimated 
effect is positive and not statistically significant but is substantively important. For more information, please refer to the WWC Procedures and Standards Handbook (version 3.0), p. 26. 


> For Castleman et al. (2014), the WWC did not need to make corrections for clustering, multiple comparisons, or to adjust for baseline differences. The p-values presented here were 
reported in the original study. Means for the intervention and comparison groups are covariate adjusted, and reflect the pooled Boston and Fulton samples. Findings for each site are 
presented in Appendix D. This study is characterized as having a statistically significant positive effect because the estimated effect is positive and statistically significant. For more 
information, please refer to the WWC Procedures and Standards Handbook (version 3.0), p. 26. 


° For Castleman et al. (2015a), no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value presented here was 
reported in the original study. Means for the intervention and comparison groups are covariate adjusted. This study is characterized as having an indeterminate effect because the 
reported effect size was neither statistically significant nor large enough to be substantially important. For more information, please refer to the WWC Procedures and Standards 
Handbook (version 3.0), p. 26. 


4 For Castleman et al. (2015b), no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value presented here was 
reported in the original study. Means for the intervention and comparison groups are covariate adjusted and reflect the pooled Shelby County and Ferguson-Florissant findings. This 
study is characterized as having an indeterminate effect because the reported effect size was neither statistically significant nor large enough to be substantially important. For more 
information, please refer to the WWC Procedures and Standards Handbook (version 3.0), p. 26. 


® For Castleman and Page (2015), no corrections for clustering or multiple comparisons and no difference-in-differences adjustments were needed. The p-value presented here was 
reported in the original study. Means for the intervention and comparison groups are covariate adjusted, and reflect the combined sample across sites. Findings for each site are 
presented in Appendix D. Sample sizes do not add up to the entire unduplicated student sample because some students were included in both the texting condition and the peer men- 
toring condition. This study is characterized as having an indeterminate effect because the reported effect size was neither statistically significant nor large enough to be substantially 
important. For more information, please refer to the WWC Procedures and Standards Handbook (version 3.0), p. 26. 
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Appendix D.1: Description of supplemental findings for the credit accumulation and persistence domain 


Mean 
(ENV E MC ri) WWC calculations 

Study Sample Intervention Comparison Mean Effect Improvement 
Outcome measure sample size group group difference size index p-value 
Castleman et al., 2014 
Continuous first-year FRL, Fulton 910 students 62.3 59.3 3.0 0.08 +3 amie) 
enrollment (%) (na) (na) 
Continuous first-year Non-FRL, 536 students 90.2 89.4 0.8 0.05 +2 > 10 
enrollment (%) Fulton (na) (na) 
Continuous enrollment into RL, Fulton 910 students 41.4 39.2 LD 0.06 +2 > 10) 
sophomore year (%) (na) (na) 
Continuous enrollment into Non-FRL, 536 students 83.4 80.9 25) 0.10 +4 > 10 
sophomore year (%) Fulton (na) (na) 
Continuous first-year ERGO) 487 students 86.5 72.6 13.9 0.53 +20 < .01 
enrollment (%) Pell eligible, (na) (na) 

Boston 
Continuous first-year ERG e=10) 177 students 88.7 85.1 3.6 0.19 +8 0) 
enrollment (%) Pell eligible, (na) (na) 

Boston 
Continuous first-year Not Pell 120 students 79.7 95.7 -16.0 -1.04 -35 < .05 
enrollment (%) eligible, (na) (na) 

Boston 
Continuous enrollment into EFC =0, 487 students 776 64.4 132 0.39 +15 < .01 
sophomore year (%) Pell eligible, (na) (na) 

Boston 
Continuous enrollment into EFC >0, 177 students 81.9 66.2 lav 0.51 +19 ant) 
sophomore year (%) Pell eligible, (na) (na) 

Boston 
Continuous enrollment into Not Pell 120 students 64.9 18.9 -14.0 -0.42 -16 all) 
sophomore year (%) eligible, (na) (na) 

Boston 


Table Notes: The supplemental findings presented in this table are additional findings from studies in this report that meet WWC design standards with or without reservations, 
but do not factor into the determination of the intervention rating. For mean difference, effect size, and improvement index values reported in the table, a positive number favors 
the intervention group and a negative number favors the comparison group. The effect size is a standardized measure of the effect of an intervention on outcomes, representing 
the average change expected for all individuals who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate 
presentation of the effect size, reflecting the change in an average individual’s percentile rank that can be expected if the individual is given the intervention. Some statistics may 
not sum as expected due to rounding. na = not applicable. FRL = free or reduced-price lunch. EFC = expected family contribution. 

For Castleman et al. (2014), no corrections for clustering and no difference-in-differences adjustments were needed. A correction for multiple comparisons was needed but did not 


affect whether any of the contrasts were found to be statistically significant. The p-values presented here were reported in the original study. Means for the intervention and compari- 
son groups are covariate adjusted. Subgroup sample sizes were obtained from an author query. 


Summer Counseling March 2018 Page 23 


WWC Intervention Report 


Appendix D.2: Description of supplemental findings for the college access and enrollment domain 


Mean 
(standard deviation) WWC calculations 
Study Sample Intervention Comparison Mean Effect Improvement 
Outcome measure sample size group group difference size index p-value 
Castleman et al., 2012 
Enrolled in 2-year Fullsample 162 students 19.0 2an0) -4.0 =Oals =(6 > 10 
institution (%) (na) (na) 
Enrolled in 4-year Fullsample 162 students 40.0 26.0 14.0 0.39 +15 Raral) 
institution (%) (na) (na) 
Castleman et al., 2014° 
Enrollment (%) Boston 927 students 83.0 78.4 46 0.18 +/ < 10 
(na) (na) 
Enrollment (%) Fulton 1,446 students 87.6 85.4 Be 0.11 +5 > 10 
(na) (na) 
Enrollment (%) FRL, Fulton 910 students TALS) 63.4 8.5 0.24 +9 < 10 
(na) (na) 
Enrollment (%) Non-FRL, 536 students 92.6 92.8 =0.2 = (02 al > 10 
Fulton (na) (na) 
Enrollment (%) ERC 0) 487 students 88.6 WO3) 123 0.53 +20 < Ol 
Pell eligible, (na) (na) 
Boston 
Enrollment (%) EFC > 0, 177 students ston 83.3 2.4 0.11 +4 > 10 
Pell eligible, (na) (na) 
Boston 
Enrollment (%) Not Pell 120 students 83.5 94.3 —10.8 a Oval —26 < 10 
eligible, (na) (na) 
Boston 
Castleman et al., 2015a° 
Enrollment (%) High school 1,063 90.9 91.8 019 —0.07 =8 = 0 
based students (na) (na) 
counseling 
(APS) 
Enrollment (%) College- 1,067 92.4 91.8 0.6 0.05 +2 > 05 
based students (na) (na) 
counseling 
(UNM) 
Enrollment (%) Hispanic 290 students 93.5 84.0 915 0.61 +23 € {0S 
males (na) (na) 
Enrollment (%) Non- 374 students 89.6 91.0 -1.4 —0.10 -4 > 05 
Hispanic (na) (na) 
males 
Enrollment (%) Hispanic 513 students 91.9 93.0 -11 —0.10 —4 = 05 
females (na) (na) 
Enrollment (%) Non- 435 students 93.4 96.0 =216 =(),32 =) > .05 
Hispanic (na) (na) 
females 
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Mean 
(standard deviation) WWC calculations 
Sample Intervention Comparison Mean Effect Improvement 
Outcome measure size group group difference size index p-value 
Castleman et al., 2015b¢ 
Enrolled in 2-year Full sample 3,281 22.9 15.4 73) 0.30 +12 < (0 
institution (%) students (na) (na) 
Enrolled in 4-year Full sample 3,281 341 36.7 AG) —0.07 =3 > 10 
institution (%) students (na) (na) 
Castleman & Page, 2015° 
Enrollment (%) Texting 2,920 14.2 71.8 24 0.07 +3 > 10 
intervention: students (na) (na) 
Dallas 
Enrollment (%) Texting 1,388 68.5 70.1 =.6 =0),05 =2 mal) 
intervention: students (na) (na) 
Boston 
Enrollment (%) Texting 746 students 69.9 62.8 Al 0.19 +8 < .05 
intervention: (na) (na) 
Lawrence & 
Springfield 
Enrollment (%) Peer 1,146 students 73.6 70.1 38) 0.10 +4 > 10 
mentoring (na) (na) 
intervention: 
Boston 
Enrollment (%) Peer 617 students 66.4 62.8 3.6 0.10 +4 > 10 
mentoring (na) (na) 
intervention 
Lawrence & 
Springfield: 
Enrollment (%) Peer 443 students 65.2 67.5 =2.3) —0.06 =2 > 10 
mentoring (na) (na) 
intervention: 
Philadelphia 
Enrolled in 2-year Texting 5,059 ZO 20.2 3.0 0.11 +4 <5) 
institution (%) intervention: students (na) (na) 
Full sample 
Enrolled in 4-year Texting 5,059 36.8 38.6 S13 —0.05 = 0) 
institution (%) intervention: students (na) (na) 
Full sample 
Enrolled in 2-year Peer 2,206 13.8 14.2 -0.4 -0.02 =| > 10 
institution (%) mentoring students (na) (na) 
intervention: 
Full sample 
Enrolled in 4-year Peer 2,206 43.3 38.8 4.5 0.11 +4 < 10 
institution (%) mentoring students (na) (na) 
intervention: 
Full sample 


Table Notes: The supplemental findings presented in this table are additional findings from studies in this report that meet WWC design standards with or without reservations, 
but do not factor into the determination of the intervention rating. For mean difference, effect size, and improvement index values reported in the table, a positive number favors 
the intervention group and a negative number favors the comparison group. The effect size is a standardized measure of the effect of an intervention on outcomes, representing 
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the average change expected for all individuals who are given the intervention (measured in standard deviations of the outcome measure). The improvement index is an alternate 
presentation of the effect size, reflecting the change in an average individual’s percentile rank that can be expected if the individual is given the intervention. Some statistics may 
not sum as expected due to rounding. na = not applicable. FRL = free or reduced-price lunch. EFC = expected family contribution. 


4 For Castleman et al. (2012), the WWC did not need to make corrections for clustering, multiple comparisons, or to adjust for baseline differences. The p-values presented here were 
reported in the original study. Means for the intervention group are covariate adjusted. The two outcomes presented here are dichotomous measures that combine three possible out- 
comes: enrollment in a 2-year institution, enrollment in a 4-year institution, and no college enrollment. For example, the enrolled in a 2-year institution variable includes both students 
who did not enroll and students who enrolled in 4-year institutions as “no” responses. 


> For Castleman et al. (2014), no corrections for clustering and no difference-in-differences adjustments were needed. A correction for multiple comparisons was needed but did not 
affect whether any of the contrasts were found to be statistically significant. The p-values presented here were reported in the original study. Means for the intervention and compari- 
son groups are covariate adjusted. Subgroup sample sizes were obtained from an author query. 


© For Castleman et al. (2015a), no corrections for clustering and no difference-in-differences adjustments were needed. A correction for multiple comparisons was needed but did not 
affect whether any of the contrasts were found to be statistically significant. The p-values presented here were reported in the original study. Means for the intervention and compari- 
son groups are covariate adjusted. Subgroup sample sizes were obtained from an author query. 


4 For Castleman et al. (2015b), no corrections for clustering and no difference-in-differences adjustments were needed. A correction for multiple comparisons was needed but did not 
affect whether any of the contrasts were found to be statistically significant. The p-values presented here were reported in the original study. Means for the intervention and compari- 
son groups are covariate adjusted. 


® For Castleman and Page (2015), no corrections for clustering and no difference-in-differences adjustments were needed. A correction for multiple comparisons was needed for the 
texting intervention but did not affect whether any of the contrasts were found to be statistically significant. The p-values presented here were reported in the original study. Means for 
the intervention and comparison groups are covariate adjusted. The two outcomes presented here are dichotomous measures that combine three possible outcomes: enrollment in a 
2-year institution, enrollment in a 4-year institution, and no college enrollment. For example, the enrolled in a 2-year institution variable includes both students who did not enroll and 
students who enrolled in 4-year institutions as “no” responses. 
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Endnotes 


1 The descriptive information for this intervention comes from: Castleman et al., 2012, Castleman et al., 2014, Castleman et al., 2015a, 
Castleman et al., 2015b, and Castleman and Page, 2015. Further verification of the accuracy of the descriptive information for this 
intervention is beyond the scope of this review. 


? The literature search reflects documents publicly available by August 2017. Reviews of the studies in this report used the standards 
from the WWC Procedures and Standards Handbook (version 3.0) and the Transition to College review protocol (version 3.2). The evi- 
dence presented in this report is based on available research. Findings and conclusions may change as new research becomes available. 


3 Please see the Transition to College review protocol (version 3.2) for a list of all outcome domains. 


4 For criteria used to determine the rating of effectiveness and extent of evidence, see the WWC Rating Criteria on p. 28. These 
improvement index numbers show the average and range of individual-level improvement indices for all findings across the studies. 


5 As cited in Castleman, Arnold, and Wartman (2012); Castleman, Page, and Schooley (2014); Castleman, Owen, and Page (2015a); 
Castleman, Owen, and Page (2015b); and Castleman and Page (2015). 
Recommended Citation 


What Works Clearinghouse, Institute of Education Sciences, U.S. Department of Education. (2018, March). 
Transition to College intervention report: Summer Counseling. Retrieved from https://whatworks.ed.gov 
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WWC Rating Criteria 
Criteria used to determine the rating of a study 


Study rating Criteria 

Meets WWC group design A study that provides strong evidence for an intervention’s effectiveness, such as a well-implemented RCT. 
standards without reservations 

Meets WWC group design A study that provides weaker evidence for an intervention’s effectiveness, such as a QED or an RCT with high 
standards with reservations attrition that has established equivalence of the analytic samples. 


Criteria used to determine the rating of effectiveness for an intervention 


Rating of effectiveness Criteria 


Positive effects Two or more studies show statistically significant positive effects, at least one of which met WWC group design 
standards for a strong design, AND 
No studies show statistically significant or substantively important negative effects. 


Potentially positive effects At least one study shows a statistically significant or substantively important positive effect, AND 
No studies show a statistically significant or substantively important negative effect AND fewer or the same number 
of studies show indeterminate effects than show statistically significant or substantively important positive effects. 


Mixed effects At least one study shows a statistically significant or substantively important positive effect AND at least one study 
shows a statistically significant or substantively important negative effect, but no more such studies than the number 
showing a statistically significant or substantively important positive effect, OR 
At least one study shows a statistically significant or substantively important effect AND more studies show an 
indeterminate effect than show a statistically significant or substantively important effect. 


Potentially negative effects One study shows a statistically significant or substantively important negative effect and no studies show 
a Statistically significant or substantively important positive effect, OR 
Two or more studies show statistically significant or substantively important negative effects, at least one study 
shows a statistically significant or substantively important positive effect, and more studies show statistically 
significant or substantively important negative effects than show statistically significant or substantively important 
positive effects. 


Negative effects Two or more studies show statistically significant negative effects, at least one of which met WWC group design 
standards for a strong design, AND 
No studies show statistically significant or substantively important positive effects. 


No discernible effects None of the studies shows a statistically significant or substantively important effect, either positive or negative. 


Criteria used to determine the extent of evidence for an intervention 


Extent of evidence Criteria 


Medium to large The domain includes more than one study, AND 
The domain includes more than one school, AND 
The domain findings are based on a total sample size of at least 350 students, OR, assuming 25 students in a class, 
a total of at least 14 classrooms across studies. 


Small The domain includes only one study, OR 
The domain includes only one school, OR 
The domain findings are based on a total sample size of fewer than 350 students, AND, assuming 25 students 
in aclass, a total of fewer than 14 classrooms across studies. 
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Glossary of Terms 


Attrition 


Baseline 


Clustering adjustment 


Confounding factor 


Design 


Effect size 
Eligibility 


Equivalence 


Attrition occurs when an outcome variable is not available for all subjects initially assigned to 
the intervention and comparison groups. If a randomized controlled trial (RCT) or regression 
discontinuity design (RDD) study has high levels of attrition, the validity of the study results 
can be called into question. An RCT with high attrition cannot receive the highest rating of 
Meets WWC Group Design Standards without Reservations, but can receive a rating of Meets 
WWC Group Design Standards with Reservations if it establishes baseline equivalence of the 
analytic sample. Similarly, the highest rating an RDD with high attrition can receive is Meets 
WWC RDD Standards with Reservations. 


For single-case design research, attrition occurs when an individual fails to complete all 
required phases or data points in an experiment, or when the case is a group and individuals 
leave the group. If a single-case design does not meet minimum requirements for phases and 
data points within phases, the study cannot receive the highest rating of Meets WWC Pilot 
Single-Case Design Standards without Reservations. 


A point in time before the intervention was implemented in group design research and in 
regression discontinuity design studies. When a study is required to satisfy the baseline 
equivalence requirement, it must be done with characteristics of the analytic sample at 
baseline. In a single-case design experiment, the baseline condition is a period during 
which participants are not receiving the intervention. 


An adjustment to the statistical significance of a finding when the units of assignment and 
analysis differ. When random assignment is carried out at the cluster level, outcomes for indi- 
vidual units within the same clusters may be correlated. When the analysis is conducted at the 
individual level rather than the cluster level, there is a mismatch between the unit of assignment 
and the unit of analysis, and this correlation must be accounted for when assessing the statisti- 
cal significance of an impact estimate. If the correlation is not accounted for in a mismatched 
analysis, the study may be too likely to report statistically significant findings. To fairly assess 
an intervention’s effects, in cases where study authors have not corrected for the clustering, the 
WWC applies an adjustment for clustering when reporting statistical significance. 


A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 


The method by which intervention and comparison groups are assigned (group design and 
regression discontinuity design) or the method by which an outcome measure is assessed repeat- 
edly within and across different phases that are defined by the presence or absence of an inter- 
vention (single-case design). Designs eligible for WWC review are randomized controlled trials, 
quasi-experimental designs, regression discontinuity designs, and single-case designs. 


The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 


A study is eligible for review and inclusion in this report if it falls within the scope of the 
review protocol and uses either an experimental or matched comparison group design. 


A demonstration that the analysis sample groups are similar on observed characteristics 
defined in the review area protocol. 
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Extent of evidence 


Gain scores 


Group design 


Improvement index 


Intervention 


Intervention report 


Multiple comparison 
adjustment 


An indication of how much evidence from group design studies supports the findings in an 
intervention report. The extent of evidence categorization for intervention reports focuses 
on the number and sizes of studies of the intervention in order to give an indication of how 
broadly findings may be applied to different settings. There are two extent of evidence cat- 
egories: small and medium to large. 


e small: includes only one study, or one school, or findings based on a total sample size 
of less than 350 students and 14 classrooms (assuming 25 students in a class) 


e medium to large: includes more than one study, more than one school, and findings 
based on a total sample of at least 350 students or 14 classrooms 


The result of subtracting the pretest from the posttest for each individual in the sample. 
Some studies analyze gain scores instead of the unadjusted outcome measure as a method 
of accounting for the baseline measure when estimating the effect of an intervention. The 
WWC reviews and reports findings from analyses of gain scores, but gain scores do not 
satisfy the WWC’s requirement for a statistical adjustment under the baseline equivalence 
requirement. This means that a study that must satisfy the baseline equivalence require- 
ment and has baseline differences between 0.05 and 0.25 standard deviations Does Not 
Meet WWC Group Design Standards if the study’s only adjustment for the baseline measure 
was in the construction of the gain score. 


A study design in which outcomes for a group receiving an intervention are compared to 
those for a group not receiving the intervention. Comparison group designs eligible for 
WWC review are randomized controlled trials and quasi-experimental designs. 


Along a percentile distribution of individuals, the improvement index represents the gain 
or loss of the average individual due to the intervention. As the average individual starts at 
the 50th percentile, the measure ranges from —50 to +50. 


An educational program, product, practice, or policy aimed at improving student outcomes. 


A summary of the findings of the highest-quality research on a given program, product, 
practice, or policy in education. The WWC searches for all research studies on an interven- 
tion, reviews each against design standards, and summarizes the findings of those that 
meet WWC design standards. 


An adjustment to the statistical significance of results to account for multiple comparisons 
in a group design study. The WWC uses the Benjamini-Hochberg (BH) correction to adjust 
the statistical significance of results within an outcome domain when study authors perform 
multiple hypothesis tests without adjusting the p-value. The BH correction is used in three 
types of situations: studies that tested multiple outcome measures in the same outcome 
domain with a single comparison group; studies that tested a given outcome measure 
with multiple comparison groups; and studies that tested multiple outcome measures in 
the same outcome domain with multiple comparison groups. Because repeated tests of 
highly correlated constructs will lead to a greater likelihood of mistakenly concluding that 
the impact was different from zero, in all three situations, the WWC uses the BH correction 
to reduce the possibility of making this error. The WWC makes separate adjustments for 
primary and secondary findings. 
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Outcome domain 
Quasi-experimental 
design (QED) 


Randomized controlled 
trial (RCT) 


Rating of effectiveness 


Regression 
discontinuity design 
(RDD) 


Single-case design 


Standard deviation 


Statistical significance 


Study rating 


Substantively important 


Systematic review 


A group of closely-related outcomes. A domain is the organizing construct for a set of 
related outcomes through which studies claim effectiveness. 


A quasi-experimental design (QED) is a research design in which study participants are 
assigned to intervention and comparison groups through a process that is not random. 


A randomized controlled trial (RCT) is an experiment in which eligible study participants are 
randomly assigned to intervention and comparison groups. 


For group design research, the WWC rates the effectiveness of an intervention in each domain 
based on the quality of the research design and the magnitude, statistical significance, and 
consistency in findings. For single-case design research, the WWC rates the effectiveness 

of an intervention in each domain based on the quality of the research design and the con- 
sistency of demonstrated effects. The criteria for the ratings of effectiveness are given in the 
WWC Rating Criteria on p. 28. 


A design in which groups are created using a continuous scoring rule. For example, stu- 
dents may be assigned to a summer school program if they score below a preset point on a 
standardized test, or schools may be awarded a grant based on their score on an applica- 
tion. A regression line or curve is estimated for the intervention group and similarly for the 
comparison group, and an effect occurs if there is a discontinuity in the two regression lines 
at the cutoff. 


A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 


The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample tend to be spread out over a large range of values. 


Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelinood that the difference is due to chance is less than 5% (p < .05). 


The result of the WWC assessment of a study. The rating is based on the strength of the 
evidence of the effectiveness of the educational intervention. Studies are given a rating of 
Meets WWC Design Standards without Reservations, Meets WWC Design Standards with 
Reservations, or Does Not Meet WWC Design Standards, based on the assessment of the 
study against the appropriate design standards. The WWC has design standards for group 
design, single-case design, and regression discontinuity design studies. 


A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


A review of existing literature on a topic that is identified and reviewed using explicit methods. 
A WWC systematic review has five steps: 1) developing a review protocol; 2) searching the 
literature; 3) reviewing studies, including screening studies for eligibility, reviewing the meth- 
odological quality of each study, and reporting on high quality studies and their findings; 

4) combining findings within and across studies; and, 5) summarizing the review. 


Please see the WWC Procedures and Standards Handbook (version 3.0) for additional details. 
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Intervention Practice 
Report Guide 


Single Study 
Review 


An intervention report summarizes the findings of high-quality research on a given program, practice, or policy in 
education. The WWC searches for all research studies on an intervention, reviews each against evidence standards, 
and summarizes the findings of those that meet standards. 


This intervention report was prepared for the WWC by Abt Associates under contract ED-IES-16-C-0024. 
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